On the Automatic Generation of Medical Imaging Reports

Medical imaging is widely used in clinical practice for diagnosis and treatment. Specialized physicians read medical images and write textual reports to narrate the findings. Report-writing can be error-prone for unexperienced physicians, and time-consuming and tedious for physicians in highly populated nations. To address these issues, we study the automatic generation of medical imaging reports, as an assistance for human physicians in producing reports more accurately and efficiently. This task presents several challenges. First, a complete report contains multiple heterogeneous forms of information, including findings which are paragraphs and tags which are a list of key words. Second, abnormal regions in medical images are difficult to identify. Generating textual narrations for them is even harder. Third, the reports are typically long, containing multiple paragraphs. To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs. We demonstrate the effectiveness of the proposed methods on a chest x-ray dataset and a pathology dataset.